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ABC's  Of  Satisfaction  Surveys 


Purpose 

A satisfaction  survey  is  a tool  that  can  be  used  by  your  agency 
to  measure  the  quality  of  your  services.  This  guide  is  being 
provided  to  assist  in  the  development  of  your  agency’s 
satisfaction  survey.  The  results  of  a survey  can  enable  your 
agency  to  measure  the  effectiveness  of  delivered  services,  and 
isolate  service  patterns  across  regions,  clients  and  staff.  The 
request  for  feedback  demonstrates  quality  to  payers  and  assists 
in  keeping  and  bringing  customers.  A satisfaction  survey  is 
also  a means  to  improve  staff  and  client  morale  and  promote 
organizational  pride. 

Your  contract  with  the  Illinois  Department  on  Aging  (IDoA) 
may  require  that  you  develop  and  use  a satisfaction  surveyfs) 
to  assess  service.  If  you  offered  to  administer  a satisfaction 
survey  when  you  applied  for  a Community  Care  Program 
(CCP)  contract  (in  your  application),  the  Department  will 
expect  to  see  the  results  of  your  effort  when  it  conducts  a 
Quality  Improvement  (QI)  review  of  your  contract.  (See  page  23 
for  more  detail  on  Department  expectations). 


Who  to  survey? 

An  agency  conducting  a satisfaction  survey  will  want  to  gather 
information  from  a variety  of  people  or  agencies  that  are 
involved  with  the  provided  services.  Clients  are  a good  source 
of  information,  but  satisfaction  surveys  do  not  have  to  be 
limited  to  only  one  group.  Other  groups  can  be  asked  to  help 
evaluate  your  agency.  You  might  consider  surveying: 
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* Clients  (current  as  well  as  past); 

* Family  (current  as  well  as  past); 

* Staff  (direct  service,  administrative); 

* Other  community  agencies  with  which  you  do 
business; 

* Records  (client,  service,  billing,  worker);  and/or 

* Visitors  (Adult  Day  Care,  Senior  Center). 


How  to  survey ? 


Method 

It  is  best  to  keep  the  survey  short,  so  that  the  respondent  takes 
no  longer  than  fifteen  minutes  to  complete  the  questionnaire. 
The  longer  the  questionnaire  and  the  more  time  it  takes  to 
complete,  the  more  likely  the  respondent  will  lose  interest. 

There  are  three  methods  to  conduct  a survey: 

* Personal  interview  - This  can  be  an  expensive 
method,  taking  a lot  of  staff  and  time.  Biased 
responses  can  result  in  a personal  interview  due  to 
face-to-face  interaction.  It  is  generally  difficult  for 
someone  to  be  completely  honest  about  his/her 
feelings,  especially  if  it  is  not  favorable,  to  an  agency 
representative.  Therefore,  responses  tend  to  be 
more  favorable  when  the  personal  interview  is  used. 

* Telephone  - Surveys  conducted  over  the  telephone 
can  also  lead  to  bias  similar  to  the  personal 
interview.  This  method  tends  to  produce  higher 
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scores  than  a survey  conducted  by  mail.  This 
method  will  also  take  a lot  of  time  but  reduces  the 
travel  time  involved  in  a personal  interview.  On 
the  other  hand,  there  may  be  phone  charges. 

* Mail  - This  method  produces  the  least  amount  of 
bias.  Surveys  that  are  sent  through  the  mail  allow 
the  respondent  to  remain  anonymous.  Mailing  the 
survey  is  more  economical  than  either  the  personal 
interview  or  telephone  survey  in  terms  of  staff  time 
involved.  However,  there  is  a cost  to  the  agency  for 
the  purchasing  postage,  envelopes,  etc.  The  fact 
that  the  return  rate  with  a mailing  is  only  1/3  to  1/2 
should  be  considered.  A low  number  of  responses 
is  another  cost  to  the  agency. 

Confidentiality  is  an  important  factor.  You  are  more  likely  to 
get  true  feelings  from  respondents  if  they  feel  they  will  remain 
anonymous.  There  may  be  hidden  fears  of  losing  service, 
service  changing,  or  some  other  type  of  retribution  due  to  any 
negative  response  they  may  want  to  express  about  the  agency. 
Assuring  confidentiality  will  help  to  elicit  honest  responses 
from  those  who  are  participating.  If  possible,  the  person 
conducting  the  survey  should  be  impartial,  for  instance,  a 
student  intern.  In  addition,  you  should  not  assign  numbers  or 
other  identifiers  to  surveys  before  they  are  distributed. 

Sample  Size 

A sample  is  the  group  of  people  who  will  be  participating  in  the 
survey.  It  should  be  representative  of  the  total  population. 
Enough  people  need  to  be  involved  in  order  to  get  different 
opinions  and  views  on  issues,  and  to  be  able  to  determine  if 
others  have  the  same  feelings  that  are  being  expressed.  The 
sample  size  is  a determining  factor  in  the  accuracy  of  the  data 
gathered.  If  your  sample  is  too  small,  the  data  will  not  give  a 
clear  picture. 
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You  will  need  to  have  a minimum  of  30  to  40  responses  from 
the  survey.  Remember,  if  mailing  a survey,  the  return  rate  is 
only  173  to  172.  Therefore,  if  you  mail  out  40  surveys,  you  may 
only  get  13  to  20  surveys  back.  If  the  population  you  are 
proposing  to  sample  is  less  than  100,  then  survey  all  (100%). 
The  general  formula  to  determine  the  percentage  to  survey  is 
100  divided  (/)  by  the  number  of  clients  multiplied  by  100  to 
equal  the  percentage.  General  formula:  100/#  of  clients  * 100 
= percentage. 

Examples:  The  population  being  surveyed  are  clients  of  the 
agency. 

You  have  a total  of  100  clients. 

100/100  clients  * 100  = 100% 

You  will  need  to  survey  all  100%  of  your  clients. 

You  have  a total  of  200  clients. 

100/200  clients  * 100  = 50% 

You  will  need  to  survey  50%  of  your  clients. 

You  have  a total  of  300  clients. 

100/300  clients  * 100  = 33% 

You  will  need  to  survey  33%  of  you  clients. 

Important  Guidelines  for  Determining  a Sample  Size 

* A minimum  of  30  to  40  responses. 

* The  return  rate  of  surveys  is  generally  1/3  to  1/2. 

* If  less  than  100,  do  all  (100%) 

* General  formula  to  determine  percentage  to  survey: 
(100/#  of  clients)  * 100  = percent 

LIBRARY  U.  OF  I.  URBANA-CHAMPA1GN 
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Selecting  a Sample 


There  should  be  a process  for  selecting  a sample  in  which  you 
would  choose  a percentage  of  your  client  population  based  on 
the  general  formula  previously  described.  People  should  not  be 
selected  to  participate  in  a survey  because  they  happen  to  be 
convenient  to  contact,  or  because  they  will  give  the  answer  you 
want  to  hear.  Remember,  a sample  is  representative  of  the 
total  population  you  want  to  survey.  For  example,  if  you  want 
to  determine  how  your  clients'  view  the  services  they  are 
receiving,  a sample  of  your  total  client  population  should  be 
surveyed.  There  are  different  methods  by  which  a sample  may 
be  selected. 

Methods  by  which  a sample  may  be  selected: 

* Random  - Sample  selection  using  this  method  is 
organized  and  structured.  Use  the  general  formula 
to  determine  the  percentage  to  survey;  if  your 
agency  has  200  clients,  you  would  want  to  survey 
50%  or  100  clients.  An  example  of  a random 
sample  would  be  using  an  alphabetical  listing  of 
all  your  200  clients  and  selecting  every  other 
person  to  participate.  Another  way  to  get  a random 
sample  would  be  to  use  a listing  of  your  client's 
social  security  numbers  and  selecting  every  other 
number.  Then  the  person  with  that  social  security 
number  would  be  survey  participant. 

* By  Time  - A set  interval  of  time  is  designated  and 
a person  in  that  time  interval  is  selected  as  a part 
of  the  sample.  Examples  of  samples  selected  by 
time  are  clients  served  on  Wednesday,  all  clients 
when  they  discontinue  service,  or  all  clients  who 
are  due  for  a redetermination  in  a particular 
month. 
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* By  Category  (Block)  - A category  is  a group  that  has 
a similar  quality.  For  example,  categories  can  be 
determined  by  a variable  such  as  sex,  time  on 
program,  impaired  level,  etc.  You  may  want  to  use 
sample  selection  by  category  when  you  have  reason 
to  be  concerned  that  certain  groups  will  think,  feel 
or  behave  differently.  For  instance,  you  may  want 
to  conduct  a survey  with  a sample  of  the  population 
who  receives  home  delivered  meals  in  addition  to 
homemaker  services.  In  this  case,  your  survey  will 
be  designed  to  explore  issues  connected  with  home 
delivered  meals.  To  select  a sample  by  category, 
first  determine  the  category.  Once  the  category  has 
been  selected,  arrange  the  people  within  the 
category,  such  as  alphabetical  order  or  by  social 
security  number.  Then  randomly  select  your 
sample  within  that  arrangement.  Remember  to  use 
the  general  formula  to  determine  the  percentage  of 
people  that  should  be  surveyed. 

Important  Guidelines  to  Selecting  a Sample 

* Random  - This  method  is  organized  and  structured , and 
achieves  the  goal  of  a representation  of  the  total 
population  surveyed. 

* By  Time  - This  method  is  based  on  selecting  a sample 
based  on  a designated  set  interval  of  time. 

* By  Category  - This  method  involves  selecting  a sample 
from  a group  with  similar  characteristics,  such  as 
gender. 
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Interview  Questions: 


Question  Content:  The  content  of  the  questions  in  a survey  is 
very  important.  How  the  reader  interprets  the  question 
determines  the  response.  Therefore,  questions  must  ask  for 
information  that  you  and  your  agency  would  like  to  explore. 
The  questions  in  a survey  should  be  specific.  The  language 
used  should  be  familiar  to  the  person  who  is  to  respond.  Try 
not  to  use  words  that  the  reader  does  not  understand.  For 
example,  you  would  not  discuss  the  laws  of  physics  with  an 
infant;  similarly,  you  should  not  use  organizational 
terminology  or  acronyms  in  a survey  designed  for  client  usage. 
Design  questions  regarding  the  issues  which  the  agency  would 
like  to  learn  about. 


© Bad  Questions:  Questions  that  are  not  specific  and 
are  free  for  individual  interpretation  are  difficult  to 
summarize  and  evaluate.  The  intent  should  be  evident 
in  order  to  get  the  information  you  are  requesting  from 
the  respondent. 

Here  are  some  examples  of  non-specific  questions: 

© Are  you  satisfied  with  your  worker? 

© Are  you  satisfied  with  your  service? 

These  are  bad  questions  because  the  client  says  yes  or 
no,  and  you  still  do  not  know  why  the  client  feels  that 
way. 


©Better  Questions:  Questions  should  be  asked  so  it  is 
clear  to  the  respondent  what  is  being  asked. 
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Here  are  some  examples  of  some  specific  questions: 

© My  worker  always  arrives  on  time?  (client  selects 
a response  ranging  from  strongly  agree  to 
strongly  disagree) 

© I have  to  remind  my  worker  what  I need  done? 
(client  selects  a response  from  strongly  agree  to 
strongly  disagree) 

© What  do  you  like  best  about  the  day  care  center? 
(client  fills  in  the  answer) 

Important  Guidelines  to  Question  Content. 

* Be  specific . 

* Language  should  be  familiar  to  the  person  being  asked 
to  respond . 

* Keep  questions  related  to  issues  regarding  what  your 
agency  would  like  to  learn. 

Responses  to  Questions:  Responses  should  be  designed  to  get 
the  most  accurate  information  possible  from  the  respondent. 
Here  are  some  examples  of  different  types  of  responses  to 
questions. 

Yes/No 

Avoid  using  yes/no  questions  unless  they  are  followed 
with  "please  describe."  Additional  information  is 
generally  needed  to  understand  the  response. 
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Example:  Question  - Has  your  worker  ever  been 
late?  Yes,  please  describe  or  No. 

Answer  - “Yes,  my  worker  was  late,  but  she  had  a 
flat  tire  and  apologized.  This  was  the  only  time  she 
was  late.” 

Qpen.Ended 

Open  ended  questions  allow  the  respondent  to  be  more 
in  depth  with  his/her  answers;  they  do  not  limit  a person 
to  a few  set  choices.  Unfortunately,  open  ended 
questions  are  harder  to  summarize. 

Example:  How  does  it  make  you  feel  if  your  worker 
is  late? 

The  respondent  has  the  opportunity  to  explain 
his/her  feelings,  e.g.  angry,  frustrated,  worried.... 

If  you  use  open-ended  questions,  the  best  way  to 
summarize  is  to: 

1.  write  down  each  different  response; 

2.  when  you  come  to  a response  similar  to  one  you 
have  written  down,  put  a hash  mark  beside  the 
written  answer;  and 

3.  the  answers  can  then  be  arranged  using  a 
frequency  distribution  (see  page  12). 
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Example:  How  does  it  make  you  feel  when  your 
worker  is  late? 


angry  1111  (4) 
scared  11  (2) 
worried  IHf  111  (8) 
do  not  care  1 (1) 


Scaled 

Scaled  responses  give  the  respondent  several  choices  on 
how  to  answer  the  question  and  ask  them  to  choose  the 
most  appropriate.  There  needs  to  be  enough  options 
get  a variation.  Remember,  some  people  are  reluctant 
use  the  most  positive  or  most  negative  response,  so  a 
three  point  scale  should  not  be  used. 

Examples  of  a scaled  response. 

5-Point  Scale 

1 = Strongly  Agree 

2 = Somewhat  Agree 

3 = Neither  Agree  or  Disagree 

4 = Somewhat  Disagree 

5 = Strongly  Disagree 


7-Point  Scale 

1 = Strongly  Agree 

2 = Moderately  Agree 

3 = Slightly  Agree 

4 = Neither  Agree  or  Disagree 

5 = Slightly  Disagree 

6 = Moderately  Disagree 

7 = Strongly  Disagree 
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Important  Guidelines  to  Question  Responses 


* Avoid  yes  I no  questions  unless  followed  by  a "please 
describe . " 

* Open  ended  questions  allow  for  more  in  depth  responses, 
but  are  harder  to  summarize. 

* Scaled  responses  give  the  respondent  several  choices,  but 
there  needs  to  be  enough  options  to  get  a variation. 

Record  Review:  Record  reviews  can  answer  important 

questions  about  service  quality  as  well  as  accuracy  of 
documentation  and  compliance  with  rules/regulations. 

Types.pf.recQrdaihat-mayjre.veal  important-information. 

* Staff  to  client  ratios 

* Attendance  at  training  sessions 

* Service  delays/interruptions 

* Absenteeism 

* Scheduling 

* Responses  to  complaints 

* Responses  to  family/client  requests 

* Health  changes  and  follow  up 

* Required  forms  and  time  frames 

* Care  plan  costs  (case  managers) 

* Referrals 

* Activity  by  worker 

* Worker  paid  hours  versus  reimbursed  service  hours 
(efficiency) 
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How  to  summarize  data ? 


Once  the  surveys  are  completed  it  is  time  to  pull  the  responses 
together  and  summarize  the  findings. 

Frequency  Distribution 

A frequency  distribution  can  be  done  by  counting  the  number 
of  times  a response  was  used  for  each  question.  A frequency 
distribution  will  show  you  how  many  people  in  the  sample 
respond  the  same  way.  Then  you  will  be  able  to  see  if 
responses  are  weighted  to  the  positive  or  negative  or  evenly 
distributed. 


Example:  My  worker  always  arrives  on  time. 


Score 

1 strongly  agree 

# Selecting 
9 

Total  % 
10% 

2 somewhat  agree 

17 

18.9% 

3 neither  agree 
or  disagree 

21 

23.3% 

4 somewhat  disagree 

33 

36.7% 

5 stronglv  disagree 

in 

11.1% 

Total 

90 

100% 

There  were  five  choices  the  respondent  was  able  to  use  to  best 
describe  his/her  feelings.  There  were  9 respondents  who 
strongly  agreed  with  the  statement,  17  respondents  somewhat 
agreed,  21  respondents  neither  agreed  nor  disagreed,  33 
respondents  somewhat  disagreed,  and  10  respondents  strongly 
disagreed.  If  you  add  the  total  number  of  responses,  it  will  give 
you  a total  number  of  people  who  responded  to  the  statement. 
The  total  in  this  example  is  90.  Now,  to  get  the  percentage  of 
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the  responses,  you  will  need  to  determine  the  number  of  people 
with  the  same  response/the  total  number  of  people  who 
responded  to  the  question. 


strongly  agree 


9/90  = 10% 


somewhat  agree 


17/90  = 18.9% 


neither  agree  or  disagree 


21/90  = 23.3% 


somewhat  disagree 


33/90  = 36.7% 


disagree 


10/90  = 11.1% 


This  shows  us  that  the  response  somewhat  disagree  had  the 
highest  percentage;  in  other  words,  36.7%  of  the  sample 
indicated  somewhat  disagree  with  the  statement  "My  worker 
always  arrives  on  time.” 

Averages 

Averages  can  be  done  by  adding  the  total  of  the  responses  to  a 
question,  then  dividing  by  the  number  of  respondents. 

Example:  My  worker  always  arrives  on  time. 

Client  Response  Score 


#1 


3 


#2 


4 


#3 


1 


#4 


3 


± 




Total  5 clients 
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15 


Explanation:  There  were  five  clients  who  responded  to  this 
statement.  Client  #1  responded  with  a 3 (neither  agree  or 
disagree),  #2  responded  with  a 4 (somewhat  disagree),  #3 
responded  with  a 1 (strongly  agree),  #4  responded  with  a 3 
(neither  agree  or  disagree),  and  #5  responded  with  a 4 
(somewhat  disagree).  Then  adding  their  responses  together 
(3+4+1+3+4  = 15)  gives  a total  of  15.  An  average  is  the  total 
response  score/the  number  of  clients.  The  average  for  this 
example  would  be  calculated  by  taking  15/5  = 3.  The  average 
score  would  then  be  3.  This  can  then  be  interpreted  that  the 
average  response  was  to  neither  agree  or  disagree. 

Often  it  is  not  enough  to  look  at  averages  alone;  looking  at 
averages  and  frequency  distributions  together  will  give  a 
clearer  picture  of  the  results.  Here  is  an  example  of  using  an 
average  and  a frequency  distribution  together  to  obtain  a better 
look  at  the  whole  picture. 

Example:  My  worker  uses  my  phone  for  personal  phone  calls. 


Average: 


Client 

#1 


Response  Score 
1 


#2  5 

#3  1 


#4  5 

#5 £L 

Total  5 clients  17 


Again  we  add  the  response  scores  (1+5+1+5+5  = 17)  to  get  the 
total  of  17.  The  average  will  be  calculated  by  taking  the 
response  score  total/number  of  clients  = the  average.  For  this 
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example  17/5  = 3.4,  and  then  3.4  would  be  considered  the 
average  score.  This  can  be  interpreted  that  the  average  score 
is  neither  agree  or  disagree.  However,  none  of  the  5 clients 
answered  neither  agree  or  disagree.  This  example  shows  that 
you  should  look  beyond  averages  for  a data  summary. 

Frequency  distribution: 


Score  itSelecting  Total  % 

1 strongly  agree  2 40% 

2 somewhat  agree  0 0 

3 neither  agree 

or  disagree  0 0 

4 somewhat  disagree  0 0 


5 strongly  disagree 3 60% 

Total  5 100% 


Explanation:  This  example  uses  the  same  method  as  the 
previous  frequency  distribution.  There  was  a total  of  5 
responses,  2 strongly  agree  and  3 strongly  disagree.  To  get  the 
percentage  you  take  the  number  of  people  who  responded  the 
same  way/the  total  number  of  people  who  responded  to  the 


question. 

strongly  agree  2/5  = 40% 

somewhat  agree  0/5  = 0 

neither  agree  or  disagree  0/5  = 0 

somewhat  disagree  0/5  = 0 

strongly  disagree  3/5  = 60% 
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This  frequency  distribution  tells  us  that  60%  of  the  people  who 
responded  to  this  statement  strongly  disagreed,  and  40%  of  the 
people  who  responded  strongly  agreed.  In  comparison  to  the 
average  using  the  same  data,  the  results  are  conflicting.  The 
average  score  is  neither  agree  or  disagree,  and  the  frequency 
distribution  shows  us  that  0 people  responded  to  the  question 
with  that  selection.  This  example  shows  the  importance  of 
using  a frequency  distribution  along  with  averages  in 
summarizing  data. 

In  this  example,  the  frequency  distribution  clearly  reveals  that 
clients  either  strongly  agree  or  strongly  disagree.  There  seems 
to  be  no  ’’middle  ground.’’  If  you  were  to  get  a result  like  this, 
you  might  begin  to  wonder  why  there  are  such  large 
differences,  and  what  is  so  different  between  the  two  groups  of 
clients. 

Using  Ranges: 

Sometimes  it  is  easier  to  group  data.  For  example,  if  you  used 
a seven  point  scale,  you  might  want  to  group  the  data  as 
follows: 

Scores  on  a 7 point  scale 

(1-2)  = strongly  agree  and  moderately  agree 

(3-4-5)  = slightly  agree,  neither  agree  or  disagree,  and 

slightly  disagree 

(6-7)  = moderately  disagree  and  strongly  disagree 
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Scores  on  a 5 point  scale 

(1-2)  = strongly  agree  and  somewhat  agree 

(3)  = neither  agree  or  disagree 

(4-5)  = somewhat  disagree  and  strongly  disagree 

This  would  group  the  respondents  who  were  in  general 
agreement,  those  who  were  "on  the  fence,"  and  those  who  were 
in  general  disagreement,  and  may  give  a clearer  picture. 
Grouping  scores  can  also  assist  in  reducing  the  bias  some  have 
against  responding  with  the  most  positive  or  most  negative 
answer  as  it  was  discussed  in  Responses  to  Questions. 

Another  type  of  range  is  a time  range.  These  are  useful  if  you 
are  grouping  clients  or  workers  by  time  in  service  or  on  the  job. 

Time  on  job: 

(0  up  to  6 mos.) 

(6  mos.  to  1 yr.) 

(1  to  5 yrs.) 

(more  than  5 yrs.) 

Comparing  Sub-groups 

Another  method  to  summarize  the  data  you  have  on  the 
completed  surveys  is  to  compare  sub-groups.  This  example 
represents  the  lengths  of  time  the  worker  has  been  on  job  and 
the  client’s  average  response  to  "my  worker  always  arrives  on 
time."  In  this  case  the  surveys  have  asked  the  client  how  long 
they  have  been  receiving  services  from  the  agency.  The 
responses  are  then  grouped  into  a timeframe;  0-6  months,  6 
months-1  year,  1-5  years,  or  more  than  5 years.  Now  with  the 
surveys  grouped  by  the  client’s  time  receiving  services,  an 
average  is  taken  for  each  group  on  the  statement,  "My  worker 
always  arrives  on  time.” 
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Example:  My  worker  always  arrives  on  time.  (This  question 
is  designed  with  a 7 point  scale  as  shown  previously). 


Time  on.J-Qh 
0-6  mos. 


Average-Client  Respenae 
5.5  (most  slightly  disagree) 


6 mos.-l  yr. 


4.2  (neither  agree  or  disagree) 


1-5  yrs. 


3.5  (most  slightly  agree) 


more  than  5 yrs. 


2.7  (most  moderately  agree) 


Explanation:  Here  you  can  see  that  the  longer  the  worker  has 

been  on  the  job,  the  more  likely  the  client  is  to  rate  the  worker 

as  always  arriving  on  time. 

Important.  GuideRm^o. Summarizing. Data 

* Frequency  distribution  examines  the  number  of  times  a 
response  is  given  for  a question . 

* Taking  averages  for  the  responses  does  not  always  give 
an  accurate  picture . The  use  of  frequency  distributions 
can  help  support  or  discount  out  averages . 

* Ranges  can  be  used  to  group  data  and  assist  to  eliminate 
bias  toward  using  strongly  agree  or  strongly  disagree. 

* Comparing  sub-groups  can  be  useful  in  establishing 
relationships. 
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How  to  evaluate  responses? 


After  the  responses  have  been  summarized,  the  results  can  be 
used  to  evaluate  systems.  The  data  can  reveal  problem  areas 
and  strengths  of  an  agency.  Particular  attention  should  be 
paid  to  high  frequency  numbers.  These  are  identifying  that  the 
concerns  or  feelings  are  wide  spread  throughout  the  population 
being  surveyed.  Comparing  sub-groups  will  help  the  agency 
determine  if  there  may  be  a relationship  between  certain  sub- 
groups and  responses.  This  enables  the  agency  to  have  insight 
to  why  problems  may  be  occurring. 

The  summarized  data  will  also  allow  for  comparisons  of 
performance  to  standards  such  as  time  from  referral  to  service 
and  time  from  appointment  to  assessment.  The  data  can  also 
be  used  to  compare  the  performance  to  state-wide  numbers  if 
available. 

Look  at  the  data  from  a variety  of  perspectives,  based  on 
hypotheses  you  have  or  can  develop.  Be  careful  about  other 
possible  explanations.  Although  you  may  have  an  explanation 
that  seems  to  be  reasonable,  there  may  be  other  circumstances 
involved.  An  example  would  be  comparing  the  length  of  time 
a worker  is  on  the  job  and  the  worker  arriving  on  time. 
According  to  the  previous  example,  it  appears  that  the  longer 
the  worker  is  on  the  job,  the  more  likely  the  worker  will  be 
there  on  time.  This  may  lead  you  to  conclude  that  if  workers 
stay  on  the  job  long  enough,  they  become  more  timely.  This 
may  not  be  the  case.  Another  explanation  could  be  that  the 
client  has  had  service  by  that  worker  for  a long  time  and 
overlooks  his/her  being  late,  or  the  client  may  fear  the 
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possibility  of  losing  that  worker  if  he/she  reports  differently  or, 
it  may  be  that  workers  who  do  not  arrive  on  time  get  fired  or 
quit.  There  may  be  instances  when  two  or  more  different 
explanations  can  result  in  the  same  findings. 

Do  not  try  to  compare  your  data  with  that  of  other  agencies. 
Even  if  questions  are  identical,  factors  in  how  the  other  agency 
administered  the  survey  may  vary.  For  instance,  one  agency 
may  have  mailed  the  survey  to  promote  confidentiality  and  the 
other  may  have  completed  personal  interviews  with  the  survey. 
The  difference  in  the  methods  used  to  conduct  a survey  could 
result  in  a noticeable  variation  in  outcomes. 

You  should  not  try  to  use  your  data  to  evaluate  individual  staff. 
Not  only  will  this  violate  your  efforts  to  respect  the 
confidentiality  of  the  persons  who  responded  to  the  survey,  but 
will  also  lead  to  biased  evaluations.  Each  supervisor  needs  to 
evaluate  staff  using  established  criteria,  not  a client 
satisfaction  survey. 


What’s  next ? 

After  the  data  has  been  evaluated,  target  one  or  more  areas  for 
improvement.  Do  not  try  to  work  on  everything  at  once.  Choose 
an  area  based  on  what  appears  to  be  the  worst  problem  or  the 
most  important  to  organizational  goals.  Try  something  to  make 
improvements  in  the  identified  area  or  areas.  Sometimes  just 
sharing  the  data  with  agency  staff  causes  a change  in  behavior. 
For  example,  if  the  survey  results  showed  a trend  throughout 
the  population  that  workers  do  not  arrive  on  time,  making  staff 
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aware  of  the  problem  may  change  their  behavior  patterns.  The 
survey  can  be  repeated,  generally  known  as  a post-test,  to 
enable  the  agency  to  compare  the  results  with  the  initial 
survey,  generally  known  as  a pre-test.  After  evaluating  the 
post-test  survey,  compare  the  responses  for  changes  or  trends. 
This  can  show  an  agency  if  the  intervention  used  to  try  and 
improve  an  area  had  any  effect. 


Important  Guidelines  to  Pre-test  and  Post-test 

* The  pre-test  is  your  first  survey. 

* Summarize  and  evaluate  data  from  the  pre-test,  target 
areas  for  improvement  and  initiate  intervention.  For 
example,  share  the  results  of  the  survey  in  a staff 
meeting). 

* Then  administer  the  second  survey,  which  is  your  post- 
test, some  time  later. 

* Look  at  the  results  of  the  pre-test  and  post-test  for  any 
trends  or  improvements. 
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Diagram 


Pre-Tett  and  Post-Teat  Scores 


This  diagram  illustrates  the  frequency  distributions  for  the 
response  ’’strongly  agree’’  and  ’’somewhat  agree"  grouped 
together  for  the  survey  statement,  "My  worker  always  arrives 
on  time"  on  both  a pre-test  and  post-test.  The  horizontal  axis 
is  the  pre-test  and  post-test.  The  vertical  axis  shows  the 
percentage  of  the  frequency  distribution  for  the  responses, 
"strongly  agree  and  somewhat  agree."  When  the  percentages 
for  the  two  responses  are  totaled  together  for  each  survey,  they 
can  be  plotted  on  the  graph  as  shown  above.  The  pre-test 
survey  had  29%  respond  strongly  agree  or  somewhat  agree  to 
the  statement  "My  worker  always  arrives  on  time."  The  post- 
test survey  had  54%  respond  strongly  agree  or  somewhat  agree 
to  the  same  statement.  This  indicates  an  improvement  in  how 
the  respondents  feel  about  their  workers’  timeliness.  The 
diagram  indicates  this  positive  trend.  As  satisfaction 
surveying  continues,  summarized  data  can  be  added  to  the 
diagram  to  show  trends  over  a period  of  time. 
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What  does  IDoAJCCP  want  to  see 
during  a QI  review? 

IDoA/CCP  will  want  to  look  at  the  whole  process  of  the  satisfaction 

survey. 

* What  youn  agency  did  in  terms  of  collecting  data. 
Reviewers  will  want  to  look  at  the  survey,  learn  who 
received  it,  how  many  respondents  there  were,  etc. 

* Summarized  data  using  frequency  distributions,  averages, 
or  comparing  sub-groups. 

* What  the  agency  decided  to  try  to  improve.  Reviewers  will 
want  to  look  at  the  areas  or  issues  identified  by  the  survey 
that  the  agency  is  working  on  or  has  worked  on  for 
improvement. 

* What  the  agency  did  to  try  to  improve.  Reviewers  will 
want  to  look  at  the  interventions  that  were  used  or  are 
going  to  be  used  by  the  agency  to  work  towards 
improvement  in  the  identified  issues  or  areas. 

i 

< 

* The  effect  of  what  the  agency  did  (change -Qy.er._  time). 
Reviewers  will  want  to  look  for  any  evidence  of  an 
improvement  or  if  the  intervention  used  created  some 
change. 
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The  Illinois  Department  on  Aging  does  not  discriminate 
in  admission  to  programs  or  treatment  of  employment  in 
programs  or  activities  in  compliance  with  the 
appropriate  State  and  Federal  statutes.  If  you  feel  you 
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Senior  HelpLine:  1-800-252-8966 
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